-
Notifications
You must be signed in to change notification settings - Fork 10
feat: add caching if prompt request fails #148
feat: add caching if prompt request fails #148
Conversation
6136af7 to
8774a00
Compare
ae555e3 to
5bcdce4
Compare
literalai/api/__init__.py
Outdated
|
|
||
| sync_api = LiteralAPI(self.api_key, self.url) | ||
| cached_prompt = self.prompt_cache.get(id, name, version) | ||
| timeout = 1 if cached_prompt else None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you could move the cache logic in the get_prompt_helper to avoid duplicating it for the sync/async versions.
literalai/api/__init__.py
Outdated
| json={"query": query, "variables": variables}, | ||
| headers=self.headers, | ||
| timeout=10, | ||
| timeout=timeout, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see the modification for the async version of make_gql_call
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is lines 1519 and 1532 in the same file
willydouhard
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks promising! I would love to see a test for it.
2f9628c to
3433ee1
Compare
7ffb0b3 to
896e303
Compare
896e303 to
3730581
Compare
|
After discussing with @clementsirieix, I'm changing the implementation to avoid a too specific or complex solution. I'm also increasing the timeout to 2 sec instead of 1. |
0bd9044 to
85c72d1
Compare
2390ad2 to
06c5047
Compare
0d93fd2 to
c5faa02
Compare
Add Prompt Caching Functionality
Changes
SharedCachePromptImplementation Details
Cache keys are generated in three formats:
idnametuple(name, version)Caching logic implemented in both sync and async
get_promptmethodsOn each successful get prompt we will cache the prompt with keys
id,name,tuple(name, version)Added fallback to cached prompts when API calls fail, with warning logs
To keep in mind this cache will be cleared each time the application Is restarted
Test Commands
Test Code
Install dependencies
Run test server
Test normal flow
curl "http://localhost:8000/prompt?name=example_prompt"Test cache with failure
First call to populate cache
curl "http://localhost:8000/prompt?name=example_prompt"Second call with simulated failure
curl "http://localhost:8000/prompt?name=example_prompt&should_fail=true"Test with ID
curl "http://localhost:8000/prompt?id=prompt_123"Test with name and version
curl "http://localhost:8000/prompt?name=example_prompt&version=1"